Initial separation of dparray into 2 parts. #184

DrTodd13 · 2020-11-25T15:13:09Z

This isn't complete since I wasn't able to get the file showing up as part of the package so I wasn't able to test much. Help needed!

PokhodenkoSA · 2020-11-30T17:30:06Z

Need tests. @vlad-perevezentsev could write tests. @DrTodd13 could you provide an idea how to test this?
Possibly dpNP had tests for dparray.

PokhodenkoSA · 2020-11-30T18:20:52Z

This PR have an issue to fix: can not import dparray from dpctl.
I see warning:

>>> from dpctl import dparray
/localdisk/work/spokhode/miniconda3/envs/dpctl-dev/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: dpctl._sycl_core.SyclDevice size changed, may indicate binary incompatibility. Expected 56 from C header, got 88 from PyObject
  return f(*args, **kwds)

Also, have an errror:

>>> import dpctl, dpctl.memory, dpctl.dparray
>>> X = dpctl.dparray.ndarray((256, 4), dtype='d')
Segmentation fault (core dumped)

PokhodenkoSA · 2020-11-30T18:21:50Z

Some tests for dparray could be taken from https://github.com/IntelPython/numba/pull/116/files#diff-74090bd7024ff78832de33f87c13b014c1646982a2c286e6200913699be5d91e.

PokhodenkoSA · 2020-11-30T18:28:29Z

Related to #188

oleksandr-pavlyk · 2020-11-30T19:54:04Z

What's the best way to push changes to this branch?

Here is where it stands now, with proposed changes applied:

In [2]: import dpctl, dpctl.memory, dpctl.dparray

In [3]: X = dpctl.dparray.ndarray((256, 4), dtype='d')

In [4]: X.__sycl_usm_array_interface__
Out[4]: {}

In [5]: X.__array_interface__
Out[5]:
{'data': (93838219988992, False),
 'strides': None,
 'descr': [('', '<f8')],
 'typestr': '<f8',
 'shape': (256, 4),
 'version': 3}

In [6]: X.base
Out[6]: <Intel(R) USM allocated memory block of 8192 bytes at 0x555869c4d000>

In [7]: X.base._pointer
Out[7]: 93838219988992

oleksandr-pavlyk · 2020-11-30T20:13:17Z

dpctl/dparray.py

+        sys.stdout.flush()
+
+import dpctl
+from dpctl._memory import MemoryUSMShared


Should be from dpctl.memory import MemoryUSMShared.

oleksandr-pavlyk · 2020-11-30T20:13:37Z

dpctl/dparray.py

+functions_list = [o[0] for o in getmembers(np) if isfunction(o[1]) or isbuiltin(o[1])]
+class_list = [o for o in getmembers(np) if isclass(o[1])]
+
+array_interface_property = "__array_interface__"


Should be __sycl_usm_array_interface__ instead.

oleksandr-pavlyk · 2020-11-30T20:14:18Z

dpctl/dparray.py

+            nelems = np.prod(shape)
+            dt = np.dtype(dtype)
+            isz = dt.itemsize
+            buf = MemoryUSMShared(nbytes=isz*max(1,nelems))


should be

nbytes = int(isz*max(1, nelems)) buf = MemoryUSMShared(nbytes)

oleksandr-pavlyk · 2020-11-30T20:15:10Z

dpctl/dparray.py

+                            dtype=dtype, buffer=buffer,
+                            offset=offset, strides=strides,
+                            order=order)
+            buf = MemoryUSMShared(nbytes=ar.nbytes)


Same as above, should be

nbytes = int(ar.nbytes) buf = MemoryUSMShared(nbytes)

oleksandr-pavlyk · 2020-11-30T20:15:35Z

dpctl/dparray.py

+        # subclass of ndarray, including our own.
+        if hasattr(obj, array_interface_property):
+            return
+        if isinstance(obj, numba.core.runtime._nrt_python._MemInfo):


numba is undefined.

@DrTodd13 Why we use this function from numba? Is it optional? Can we copy this function to our code?

oleksandr-pavlyk · 2020-11-30T20:26:02Z

Method __call__ is handled in __array_ufunc__, but method __reduce__ is not:

In [7]: ms = dpctl.memory.MemoryUSMShared(8*4)

In [8]: X = dpctl.dparray.ndarray((4,), dtype='d', buffer=ms)

In [9]: X
Out[9]:
ndarray([4.64247707e-310, 4.64247698e-310, 9.48606040e-322,
         2.37151510e-322])

In [10]: np.add(X, 1)
Out[10]: ndarray([1., 1., 1., 1.])

In [11]: np.add(X, 1).base
Out[11]: <Intel(R) USM allocated memory block of 32 bytes at 0x5575e19ea000>

In [12]: X.sum()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-12-16f51ebe4109> in <module>()
----> 1 X.sum()

/home.conda/envs/idp/lib/python3.7/site-packages/numpy/core/_methods.py in _sum(a, axis, dtype, out, keepdims, initial, where)
     36 def _sum(a, axis=None, dtype=None, out=None, keepdims=False,
     37          initial=_NoValue, where=True):
---> 38     return umr_sum(a, axis, dtype, out, keepdims, initial, where)
     39
     40 def _prod(a, axis=None, dtype=None, out=None, keepdims=False,

TypeError: operand type(s) all returned NotImplemented from __array_ufunc__(<ufunc 'add'>, 'reduce', ndarray([4.64247707e-310, 4.64247698e-310, 9.48606040e-322,
         2.37151510e-322]), axis=None, dtype=None, keepdims=False, where=True): 'ndarray'

PokhodenkoSA · 2020-11-30T20:26:04Z

@oleksandr-pavlyk I will fix all your review and notes tomorrow. This PR is in the core of many further modifications and I would like to merge it tomorrow after fixing and tests adding.

PokhodenkoSA · 2020-12-02T13:28:21Z

Method __call__ is handled in __array_ufunc__, but method __reduce__ is not:

@oleksandr-pavlyk @DrTodd13 How it could be fixed? Is it ok to create a separate issue for it and fix in separate PR?

Initial separation of dparray into 2 parts.

d2458eb

This was referenced Nov 30, 2020

Remove dparray from dpNP IntelPython/dpnp#322

Closed

Initial support for Numpy subclasses (for dparray) IntelPython/numba#116

Closed

oleksandr-pavlyk self-requested a review November 30, 2020 19:25

oleksandr-pavlyk mentioned this pull request Nov 30, 2020

dpctl.memory.MemoryUSMShared(np.int64(16)) should work #189

Closed

oleksandr-pavlyk reviewed Nov 30, 2020

View reviewed changes

This was referenced Nov 30, 2020

Using __array_interface__ instead of adding __sycl_usm_array_interface__ #191

Closed

Add dparray #192

Merged

diptorupd merged commit 6b892ea into IntelPython:master Dec 7, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Initial separation of dparray into 2 parts. #184

Initial separation of dparray into 2 parts. #184

Uh oh!

DrTodd13 commented Nov 25, 2020

Uh oh!

PokhodenkoSA commented Nov 30, 2020

Uh oh!

PokhodenkoSA commented Nov 30, 2020 •

edited

Loading

Uh oh!

PokhodenkoSA commented Nov 30, 2020

Uh oh!

PokhodenkoSA commented Nov 30, 2020

Uh oh!

oleksandr-pavlyk commented Nov 30, 2020 •

edited

Loading

Uh oh!

oleksandr-pavlyk Nov 30, 2020

Uh oh!

oleksandr-pavlyk Nov 30, 2020

Uh oh!

oleksandr-pavlyk Nov 30, 2020

Uh oh!

oleksandr-pavlyk Nov 30, 2020

Uh oh!

oleksandr-pavlyk Nov 30, 2020

Uh oh!

PokhodenkoSA Dec 1, 2020

Uh oh!

oleksandr-pavlyk commented Nov 30, 2020 •

edited

Loading

Uh oh!

PokhodenkoSA commented Nov 30, 2020

Uh oh!

PokhodenkoSA commented Dec 2, 2020

Uh oh!

Uh oh!

Initial separation of dparray into 2 parts. #184

Initial separation of dparray into 2 parts. #184

Uh oh!

Conversation

DrTodd13 commented Nov 25, 2020

Uh oh!

PokhodenkoSA commented Nov 30, 2020

Uh oh!

PokhodenkoSA commented Nov 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PokhodenkoSA commented Nov 30, 2020

Uh oh!

PokhodenkoSA commented Nov 30, 2020

Uh oh!

oleksandr-pavlyk commented Nov 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oleksandr-pavlyk Nov 30, 2020

Choose a reason for hiding this comment

Uh oh!

oleksandr-pavlyk Nov 30, 2020

Choose a reason for hiding this comment

Uh oh!

oleksandr-pavlyk Nov 30, 2020

Choose a reason for hiding this comment

Uh oh!

oleksandr-pavlyk Nov 30, 2020

Choose a reason for hiding this comment

Uh oh!

oleksandr-pavlyk Nov 30, 2020

Choose a reason for hiding this comment

Uh oh!

PokhodenkoSA Dec 1, 2020

Choose a reason for hiding this comment

Uh oh!

oleksandr-pavlyk commented Nov 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PokhodenkoSA commented Nov 30, 2020

Uh oh!

PokhodenkoSA commented Dec 2, 2020

Uh oh!

Uh oh!

PokhodenkoSA commented Nov 30, 2020 •

edited

Loading

oleksandr-pavlyk commented Nov 30, 2020 •

edited

Loading

oleksandr-pavlyk commented Nov 30, 2020 •

edited

Loading